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Abstract 


To  date,  most  of  the  fault-tolerant,  real-time  systems  have  been  implemented  in  embedded  set¬ 
tings,  and  there  is  an  urgent  need  to  open  up  this  type  of  computing  technology  to  a  larger 
number  of  people  who  use  heterogeneous  distributed  computing  environments.  Today’s  trans¬ 
portation,  manufacturing,  and  communication  systems  require  the  integration  of  multiple 
embedded  real-time  control  systems  with  standard  distributed  computing  environments  in  a 
predictable  fashion.  Humboldt  University  has  developed  the  concept  of  composite  objects  as  a 
filtering  bridge  between  standard  middleware  platforms  and  software  frameworks  providing 
services  with  certain  quality-of-service  (QoS)  guarantees.  Current  research  focuses  on  the 
Common  Object  Request  Broker  Architecture  (CORBA)  middleware  platform;  however,  com¬ 
posite  objects  are  also  applicable  to  platforms  like  the  Distributed  Component  Model  (DCOM) 
and  distributed  computing  environments  (DCEs).  Key  concepts  in  Humboldt’s  approach  are 
analytic  redundancy,  noninterference,  interoperability,  and  adaptive  abstraction.  These  con¬ 
cepts  originated  in  SEI  work  on  the  Simplex  architecture  and  have  been  reapplied  to  extend 
the  reach  of  commercial  off-the-shelf  (COTS)  software  technologies  into  demanding  applica¬ 
tion  settings  (such  as  those  found  in  military  and  industrial  applications).  Here,  we  discuss 
building  blocks  and  techniques  for  fault-tolerant,  real-time  applications  based  on  CORBA. 
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1  Introduction 


Most  of  the  fault-tolerant,  real-time  systems  have  been  implemented  in  embedded  settings, 
and  there  is  an  urgent  need  to  open  up  this  type  of  computing  technology  to  a  larger  number  of 
people  who  use  heterogeneous  distributed  computing  environments  [Soley  98]. 

The  Object  Management  Group’s  (OMG)  Common  Object  Request  Broker  Architecture 
(CORBA)  is  an  important  and  popular  technology  that  supports  the  development  of  object- 
based,  distributed  applications.  The  benefits  of  abstraction  promised  by  CORBA  (location 
transparency,  heterogeneity,  dynamic  configuration,  etc.)  are  appealing  in  many  application 
domains,  including  those  that  satisfy  real-time  and  fault-tolerance  requirements  (such  as  man¬ 
ufacturing,  process  control,  and  transport  systems).  However,  CORBA  is  focused  on  facilitat¬ 
ing  general  computing  environments,  and  the  specification  of  timing  behavior  and  quality-of- 
service  (QoS)  parameters,  such  as  communication  latency  and  acceptable  processor  utiliza¬ 
tion,  is  beyond  the  scope  of  the  latest  version  of  CORBA. 

The  Computer  Architecture  and  Organization  Group  at  Humboldt  University  is  studying 
numerous  aspects  relating  to  responsive  computing  (in  particular,  techniques  for  timely  deliv¬ 
ery  of  computing  results  even  in  the  face  of  faults).  In  the  context  of  this  work,  the  Responsive 
CORBA  Unified  Environment  (RESCUE)  Project  is  developing  a  CORBA-based  distributed 
framework  for  responsive  (fault-tolerant,  real-time)  services,  which  exploits  consensus  for 
synchronization,  reliable  communication,  and  fault  diagnosis  among  replicated  server  objects. 
The  technical  foundation  for  RESCUE  is  provided  by  composite  objects,  which  act  as  a  filter¬ 
ing  bridge  (see  Figure  1-1)  between  CORBA  (which  does  not  provide  quality-of-service  guar¬ 
antees)  and  responsive  services  (which  do  provide  such  guarantees).  Composite  objects 
provide  CORBA  clients  with  higher  predictability  regarding  timely  and  reliable  method 
execution. 

Here,  we  present  the  composite  objects  approach  for  predictable  integration  of  CORBA  with 
real-time  requirements.  We  discuss  data  replication  and  weak  memory  consistency  as  the  key 
concepts  for  implementing  the  composite  objects  approach.  We  evaluate  the  timing  behavior 
of  composite  objects  and  demonstrate  the  value  of  our  technique  in  a  number  of  proof-of-con- 
cept  scenarios. 
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Figure  1-1:  Bridging  Middleware  and  QoS-Sensitive  Services  with  Composite 
Objects 

The  rest  of  this  report  is  organized  as  follows:  Section  2  discusses  the  composite  objects 
approach  for  predictable  integration  of  CORBA  and  real-time  fault-tolerant  computing.  Sec¬ 
tion  3  presents  the  problems  of  the  producer/consumer/viewer  and  the  “unstoppable  robots” 
models  and  evaluates  their  timing  behavior.  In  Section  4,  we  outline  our  key  concepts  for 
CORBA-based  fault-tolerant  computing  and  present  example  scenarios  for  fault-tolerant 
Netscape  and  the  Java-based  fault-tolerant  maze.  In  Section  5,  we  summarize  related  work 
with  the  objective  of  extending  CORBA  with  real-time  and  fault-tolerance  capabilities. 
Finally,  we  present  our  conclusions  and  some  directions  for  future  work  in  Section  6. 
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2  Composite  Objects:  Making  Tradeoffs 
Explicit 


The  composite  objects  approach  allows  the  programmer  to  make  an  explicit  tradeoff  between 
an  application’s  predictable  resource  utilization  and  its  communication  latency.  Therefore, 
composite  objects  make  implementation  details  that  are  usually  hidden  and  abstracted  away  by 
CORBA  visible  and  explicit. 

The  implementation  of  the  composite  objects  approach  follows  three  basic  design  rules: 

1 .  noninterference:  General-purpose  computing  and  responsive  computing  should  not  bur¬ 
den  each  other. 

2.  interoperability:  Services  exported  by  general-purpose  computing  objects  and  by  respon¬ 
sive  computing  objects  can  be  used  by  each  other. 

3.  adaptive  abstraction:  Lower  level  information  and  scheduling  details  are  available  for 
real-time  objects,  but  are  transparent  to  non-real-time  objects. 

Since  composite  objects  provide  functionality  to  react  on  CORBA  method  invocations,  they 
can  be  seen  as  descendants  of  a  class  that  implements  object  adaptor  functionality  (_skel  in 
many  implementations).  On  the  other  hand,  composite  objects  have  the  capability  to  create 
real-time  programming  abstractions  such  as  prioritized  threads  and  real-time  communication 
channels.  As  depicted  in  Figure  2-1,  they  can  be  seen  as  descendants  of  a  class  that  imple¬ 
ments  real-time  servers. 

Composite  objects  consist  of  a  real-time  part  and  a  non-real-time  part.  Design  time  and  run¬ 
time  guarantees  can  be  given  for  execution  of  real-time  methods.  In  contrast,  methods  in  the 
non-real-time  part  are  executed  following  a  best-effort  approach.  Composite  objects  establish 
timing  firewalls  [Poize  97]  between  real-time  and  non-real-time  (CORBA)  computing,  so  that 
the  non-real-time  part  cannot  violate  the  real-time  scheduling  rules  that  are  needed  by  the  real¬ 
time  part  [Sha  94]. 
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Figure  2-1:  Structure  of  a  Composite  Object 


Data  replication  is  the  key  to  independent  data  access  from  real-time  and  non-real-time  threads 
inside  a  composite  object.  We  split  an  object’s  data  into  two  parts:  one  that  is  statically  locked 
in  memory  to  fulfill  real-time  scheduling  assumptions  (RT  data),  and  another  that  is  treated 
like  a  standard  non-real-time  user’s  process  data  (NRT  data)  and  is  subject  to  memory  man¬ 
agement  such  as  paging.  A  pair  of  RT/NRT  variables  can  be  viewed  as  a  replicated  variable. 
Composite  objects  implement  a  consistency  protocol  to  update  both  replicas  and  create  the 
impression  of  a  shared  variable. 

The  consistency  protocol  for  shared  RT/NRT  variables  as  well  as  resource  allocation  [central 
processing  unit  (CPU)  cycles,  memory]  and  call  admission  for  CORBA  clients  are  crucial  for 
implementing  the  concept  of  composite  objects.  Based  on  the  Mach  (NeXTSTEP  3.3)  and 
rtLinux  [Barabanov  97]  operating  systems,  we  have  used  pipes,  shared  buffers,  and  message 
queues  (Mach  IPC)  for  our  implementation  of  the  composite  objects  approach  as  shown  in 
Figure  2-2. 

Figure  2-2  describes  the  data  flow  during  read/write  accesses  to  a  composite  object’s  vari¬ 
ables.  read  requests  in  both  parts  of  the  composite  object  are  always  satisfied  by  accessing  the 
local  copy  of  a  replicated,  shared  RT/NRT  variable.  In  contrast,  write  requests  not  only  result 
in  updating  the  local  copy,  but  the  value  is  also  stored  in  an  interprocess  communication  (IPC) 
data  structure  (i.e.,  pipes,  rt-fifos).  Handler  threads  periodically  copy  values  out  of  the  IPC 
data  structures  into  the  local  replica  of  the  shared  RT/NRT  variable.  The  update  rate  for  those 
handler  threads  is  programmable. 
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Figure  2-2:  Consistency  Protocol  and  Communication  Inside  a  Composite  Object 


Alternatives  to  the  implementation  outlined  here  include  use  of  shared  memory  or  message 
passing  (Mach  IPC)  for  data  transfer  between  the  CORBA  part  and  real-time  part  of  the  com¬ 
posite  object.  In  our  experiments,  we  have  evaluated  implementations  of  the  composite  objects 
approach  based  on  pipes  and  shared  memory. 

Another  design  alternative  is  an  event-triggered  consistency  protocol  in  contrast  to  the  time- 
triggered,  periodic  protocol  described  above.  In  that  case,  a  write  operation  would  include  sig¬ 
naling  the  arrival  of  a  new  data  value  to  the  handler  thread  on  the  opposite  side  of  the  compos¬ 
ite  object.  We  have  implemented  consistency  protocols  following  both  schemes. 

To  evaluate  the  overhead  introduced  by  the  composite  objects  approach,  we  have  investigated 
a  scenario  where  a  real-time  data  source  is  accessed  by  a  CORBA  client  through  a  composite 
object  that  implements  an  event-triggered  consistency  protocol. 

Figure  2-3  shows  communication  delays  for  the  described  scenario.  The  lower,  dotted  curve 
represents  the  latency  of  real-time  communication  between  the  composite  object  and  the  data 
source  (Mach  Interprocess  Communication).  The  middle  curve  shows  the  time  needed  for  the 
consistency  protocol  inside  the  composite  object  plus  real-time  communication.  The  distance 
between  the  lower  and  middle  curves  represents  the  overhead  imposed  by  our  implementation 
of  the  composite  objects  approach.  It  is  roughly  1  ms  (millisecond)  (10%  of  the  overall  com¬ 
munication  latency).  One  should  notice  that  both  curves  are  fairly  stable,  indicating  that  intro¬ 
duction  of  a  composite  object  as  a  mediator  will  not  disturb  an  application’s  predictable  timing 
behavior. 
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Figure  2-3:  Overheads  Imposed  by  Composite  Objects 


In  contrast,  the  curve  at  the  top  of  the  diagram,  which  represents  the  overall  communication 
delay  including  CORBA,  shows  clearly  visible  peaks.  Although  the  average  CORBA  commu¬ 
nication  latency  is  around  12  ms  (the  client  was  run  remotely),  even  in  the  unloaded  case  it 
varies  occasionally  by  one  order  of  magnitude.  This  indicates  that  the  direct  execution  of 
CORBA  calls  from  within  a  time-critical  application  will  significantly  disturb  the  applica¬ 
tion’s  predictability. 
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Real-Time  Case  Studies 


3.1  Java  Interface  to  a  Soft  Real-Time  Legacy 
Application 

As  a  proof-of-concept  vehicle,  we  have  created  a  responsive  service  by  extending  a  CORE/ 
SONiC  (Concensus  for  REsponsiveness/Shared  Objects  Net-interconnected  Computer)  soft 
real-time  legacy  application  with  CORBA  interfaces  that  are  accessed  by  two  Java  applets. 

The  “balancing  robots”  [Werner  96]  (shown  in  Figure  3-1)  simulate  the  following  scenario: 
Robots  are  moving  on  top  of  a  plate  that  is  kept  in  unstable  balance.  Robots  are  constantly 
using  energy  and,  therefore,  occasionally  have  to  visit  a  fuel  station,  which  is  located  outside 
of  the  plate’s  center.  Consensus  among  robots  is  achieved  prior  to  every  move,  with  robots  that 
have  high  levels  of  fuel  acting  as  a  counterbalance  to  those  moving  toward  the  fuel  station. 
Collision  avoidance  is  also  implemented  by  consensus. 


Figure  3-1:  Balancing  Robots  Simulation 


The  application  demonstrates  tolerance  of  the  robots’  and  controllers’  crash,  omission,  and 
computation  faults.  Control  is  subject  to  soft  real-time  constraints.  Calculations  of  the  control¬ 
lers’  and  robots’  moves  are  executed  with  a  frequency  of  4  Hz.  The  application  of  composite 
objects  in  a  scenario  with  more  stringent  timing  requirements  is  given  in  Section  3.2. 

Using  the  composite  objects  technique,  we  have  modified  two  of  the  application’s  interfaces  to 
use  CORBA.  Using  the  Java  language  binding  for  CORBA,  one  applet  has  been  implemented 


CMU/SEI-99-TR-001 


7 


to  monitor  the  soft  real-time  simulation  and  periodically  obtain  status  information.  Another 
applet  implements  the  robots’  control  algorithm  in  Java  and  interacts  with  the  real-time  simu¬ 
lation’s  native  controllers  during  consensus  rounds.  These  interactions  have  fixed  deadlines 
(inner  control  loop)  that  must  be  met  despite  CORBA’s  and  Java’s  variations  in  communica¬ 
tion  latency  and  execution  times.  Both  Java  applets  are  connected  to  the  “balancing  robots”  via 
gateways  implemented  as  composite  objects.  Figure  3-2  shows  the  communication  structure  of 
the  complete  application. 


algorithm 


Figure  3-2:  Communication  Structure  of  the  Balancing  Robots 


Problematic  in  our  scenario  is  the  potentially  varying  number  of  clients  (Java  applets)  access¬ 
ing  the  service.  This  may  result  in  an  unacceptably  high  load  on  the  real-time  system  and,  in 
turn,  missed  deadlines  (indicated  by  stopped/crashed  robots  that  ran  out  of  fuel).  To  circum¬ 
vent  these  problems,  we  use  the  following  three  strategies  inside  the  composite  objects  bridg¬ 
ing  between  CORBA  and  the  soft  real-time  application: 

1 .  caching:  Due  to  the  periodic  operation  of  the  soft  real-time  system,  it  is  sufficient  to  read 
status  information  once  per  period.  Thus,  the  composite  object  serves  as  a  data  cache, 
whose  contents  are  valid  for  a  limited  time  (the  length  of  one  period).  This  results  in  a 
bounded  load  for  the  soft  real-time  simulation. 

2.  call  admission:  Even  when  using  the  caching  approach  mentioned  above,  incoming 
CORBA  requests  place  some  load  on  the  real-time  system  (e.g.,  scheduling  overheads, 
processing  of  network  protocols).  To  bound  these  loads,  the  CORBA  ORB’s  dispatch  rou- 
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tine  can  be  executed  under  control  of  the  scheduling  server  [Poize  97]  effectively  restrict¬ 
ing  its  percentage  of  CPU  usage.  We  have  used  this  technique  to  implement  call  admission 
within  the  composite  objects  used  by  the  application  in  our  examples. 

3.  fallback  procedures:  The  Java-based  controller  allows  the  interactive  user  to  forward 
input  to  the  real-time  simulation.  However,  variations  in  communication  and  execution 
times  due  to  CORBA  and  Java  affect  the  soft  real-time  simulation’s  inner  control  loop  and 
may  result  in  missed  deadlines.  We  have  used  the  technique  of  fallback  procedures 
depicted  in  Figure  3-3  to  deal  with  those  cases.  In  our  approach,  the  fallback  procedure  is 
invoked  simultaneously  with  the  CORBA  invocation  of  the  Java  control  algorithm.  In  the 
case  of  a  late  reply,  the  fallback  procedure’s  result  is  used  instead  of  the  Java  algorithm’s. 
However,  in  contrast  to  the  Java  algorithm,  the  fallback  procedure’s  resource  requirements 
and  execution  times  are  known  and,  therefore,  can  be  factored  into  the  real-time  applica¬ 
tion’s  scheduling  analysis. 


Figure  3-3:  Java  Controller:  Execution  Scheme  with  Fallback  Procedure 


In  Figure  3-4,  we  compare  the  latency  of  CORBA  requests  sent  to  the  Java-based  controller 
with  the  “balancing  robots”  simulation’s  period.  We  see  that  when  an  unloaded  system  runs 
the  Java  controller,  CORBA  latency  is  well  below  the  real-time  simulation’s  period  in  most 
cases.  Even  on  occasional  delayed  calls,  the  result  is  available  before  the  simulation’s  next 
period  begins. 

In  contrast,  when  a  loaded  system  runs  the  Java  controller,  execution  results  are  often  late. 
Without  applying  a  fallback  procedure,  the  “balancing  robots”  behavior  would  have  been  sig¬ 
nificantly  disturbed.  However,  our  experiments  indicate  stable  behavior  of  the  simulation  and 
demonstrate  the  ability  of  our  technique  to  deal  with  variations  in  communication  and  execu¬ 
tion  times  typical  for  middleware-based  distributed  computing  environments. 
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The  “balancing  robots”  Web-service  has  been  implemented  on  top  of  the  Mach-based  NeXT- 
STEP  3.3  operating  system.  On  NeXTSTEP  we  have  used  the  Inter-Language  Unification 
(ILU)  2.0  alpha  12  CORBA  implementation,  whereas  the  Java-applet  connecting  to  our  Web 
service  has  been  implemented  based  on  the  Java  object  request  broker  (ORB)  of  OmniBroker 
[OOC  99].  Figure  3-5  shows  screen  dumps  of  the  Java  components. 
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Figure  3-4:  Execution  of  Java  Controller  Versus  Simulation’s  Period 
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Figure  3-5:  Screen  Dump  of  Java  Components  for  the  Balancing 
Robots 
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3.2  Producer/Consumer/Viewer  Example 
Application 

In  this  section,  we  describe  the  effects  and  applicability  of  the  composite  objects  approach  in  a 
minimal  application  setting.  Our  producer/consumer  example  studies  the  timing  behavior  of 
two  threads  that  communicate  via  shared  memory.  Both  threads  are  scheduled  according  to  the 
fixed-priority  scheduling  policy  available  on  the  NeXTSTEP  operating  system.  The  example 
can  be  seen  as  a  generic  real-time  application,  where  processor  and  resource  requirements  are 
known. 

The  producer/consumer  application  scheduled  with  fixed  priority  is  a  good  example  to  study 
CORBA’s  impact  on  the  predictability  of  an  application’s  behavior.  Both  the  producer  and 
consumer  run  periodically  and  go  through  several  phases.  Both  threads  do  some  real  work 
(produce/consume  data  for  periods  Tp/Tc,  respectively),  then  both  threads  try  to  access  the 
shared  memory  buffer  (read/write  for  periods  T/T^  respectively),  and  finally  they  sleep  (for 
period  Ts)  until  their  next  invocation.  In  our  example,  the  theoretical  execution  times  for  both 
producer  and  consumer  are  60  ms  per  invocation. 

Access  to  the  shared  memory  buffer  is  guarded  by  a  mutex.  Since  the  buffer  is  bounded,  two 
condition  variables  are  used  to  synchronize  the  producer  and  consumer.  The  resource  require¬ 
ments  and  timing  behavior  of  both  the  producer  and  consumer  are  simulated  by  computation¬ 
intensive /or-loops,  while  waiting  for  the  next  invocation  to  be  implemented  using  the  selectQ 
system  call.  Figure  3-6  shows  the  overall  scenario  and  gives  the  theoretical  parameters  for  the 
threads’  execution  times. 


Viewer 
acting  as 
CORBA  Server 
(version  A) 


CORBA  call 


Producer 


produce  -  Tp  =  32.5ms 
write  to  buffer  -  TW  =  13ms 
sleep  -Ts  =  14.5ms 


Viewer 
acting  as 
CORBA  Client 
(version  B) 


TL  Consumer 

Y 

read  from  buffer  -  Tr  =  6.5ms 
consume  -  Tc  -  32.5ms 
sleep  -Ts- 21ms 


Figure  3-6:  Producer/Consumer/Viewer:  Overall  Scenario 
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In  addition  to  producer  and  consumer,  Figure  3-6  depicts  two  variants  of  a  third  component — 
the  viewer.  We  assume  that  the  data  which  are  communicated  by  the  producer  and  consumer 
are  of  interest  to  some  kind  of  graphical  display,  which  is  connected  via  a  composite  object 
and  CORBA.  In  all  experiments,  the  viewer  component  has  been  run  remotely. 

In  general,  there  are  two  approaches  to  attach  new  CORBA  components  to  a  legacy  applica¬ 
tion:  (1)  The  application  may  either  hand  out  data  on  request  (i.e.,  act  as  a  CORBA  server,  ver¬ 
sion  A  in  Figure  3-6),  or  (2)  it  may  actively  send  data  to  the  CORBA  components  (i.e.,  act  as  a 
CORBA  client,  version  B  in  Figure  3-6).  In  the  first  case,  resource  problems  resulting  from 
varying  numbers  of  clients  requesting  data  may  show  up.  We  want  to  study  both  cases  and 
evaluate  the  timing  behavior  of  our  original  application.  We  measure  predictability  of  our 
application  in  terms  of  changes  in  the  producer’s  and  consumer’s  periodicity.  Ultimately,  we 
are  going  to  demonstrate  how  the  composite  objects  technology  may  keep  the  producer/con¬ 
sumer’s  behavior  predictable — despite  of  and  without  changes  to  CORBA. 

For  our  experiments,  we  have  used  the  Mach-based  NeXTSTEP  3.3  operating  system  run  on  a 
Hewlett-Packard  (HP)  715/50  computer  as  host  for  producer  and  consumer.  We  have  run  the 
viewer  process  remotely  on  a  SparcStation~2  computer  with  the  Solaris  2.6  operating  system. 
On  the  NeXTSTEP  side,  we  have  used  the  XEROX  PARC  Inter-Language  Unification  (ILU 
2.0  alphal2)  implementation,  which  provides  most  features  specified  in  the  CORBA  standard 
and  is  interoperable  with  CORBA-compliant  object  request  brokers.  We  have  used  Object  Ori¬ 
ented  Concept’s  (OOC’s)  OmniBroker  2.02  CORBA  implementation  on  the  Solaris  system. 


3.2.1  Version  A:  Composite  Object  as  CORBA  Client;  Viewer 
as  CORBA  Server 

In  this  scenario,  the  viewer  component  acts  as  CORBA  server,  and  our  “legacy”  real-time 
application  sends  data  via  CORBA.  Figure  3-7  compares  variation  in  the  producer/consumer’s 
periodicity  in  the  initial  case  and  for  a  naive  solution,  where  the  consumer  directly  calls  the 
viewer  (without  the  intermediate  composite  object). 

The  right-hand  diagram  in  Figure  3-7  demonstrates  CORBA’s  varying  communication  laten¬ 
cies  under  changing  loads.  In  our  experiments  we  have  run  multiple  computation-intensive 
processes  on  the  viewer’s  host  computer  to  simulate  a  slow  CORBA  server  on  a  loaded  sys¬ 
tem.  However,  even  in  the  case  of  a  fast,  unloaded  CORBA  server,  one  may  notice  substantial 
variations  in  the  producer/consumer’s  periodicity,  which  are  caused  by  CORBA.  Therefore, 
we  have  introduced  a  composite  object  as  a  mediator  between  the  producer/consumer  and 
viewer+CORBA. 
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Producer/Consumer/Viewer  -  direct  communication  via  CORBA 

CORBA  calls  executed  directly  by  consumer 
Variance:  {fast  Server)  0.101054 


Initial  version  -  no  CORBA,  no  Viewer  thread 
Variance:  0.056666 


number  of  executions 


number  of  executions 


Figure  3-7:  How  CORBA  Affects  a  Client  Application's  Timing 
Behavior 


The  variation  coefficient  is  given  in  all  diagrams  as  a  measure  for  our  test  application’s  stable 
timing  behavior.  All  diagrams  have  been  normalized  to  allow  easy  comparison  with  the  initial 
unloaded  test  run  presented  in  Figure  3-7.  In  all  CORBA-related  diagrams,  we  compare  the 
case  of  an  unloaded  remote  computer  running  the  CORBA  server  (fast  server)  against  a  loaded 
remote  computer  (slow  server). 

Figure  3-8  compares  two  implementations  of  the  composite  objects  approach.  First,  we  have 
used  shared  memory  for  internal  communication  inside  the  composite  object  (left  diagram  in 
Figure  3-8).  In  contrast,  the  right  diagram  in  Figure  3-8  shows  the  application’s  timing  behav¬ 
ior  when  pipes  are  used  to  implement  the  composite  object’s  shared  RT/NRT  variables.  In  this 
case,  the  producer  thread  directly  writes  data  onto  a  shared  RT/NRT  variable  of  the  composite 
object  and,  simultaneously,  into  the  pipe. 

Both  diagrams  in  Figure  3-8  represent  a  timing  behavior  for  the  producer/consumer  applica¬ 
tion,  which  is  quite  similar  to  the  original  unloaded  case  shown  in  the  previous  figure.  Thus, 
using  the  composite  objects  technique,  we  were  able  to  decouple  CORBA  communication 
with  the  viewer  component  from  the  original  real-time  application  and  to  save  the  applica¬ 
tion’s  timing  behavior.  Further  studies  will  compare  results  achieved  on  Mach 
(NeXTSTEP3.3)  with  a  similar  setting  on  the  rtLinux  operating  system. 


CMU/SEI-99-TR-001 


13 


Producer/Consumer/Viewer  -  Composite  Object  &  CORB  A 


Internal  communication  via  shared  memory  &  separate  thread  Internal  communication  via  pipe 
Variance:  (fast  Server)  0.044531  Variance:  (fast  Server)  0.061454 

(slow  Server)  0.056601  (slow  Server)  0.063903 


Figure  3-8:  Decoupling  CORBA  and  the  Application  via  a  Composite 
Object 


3.2.2  Version  B:  Composite  Object  as  CORBA  Server; 

Viewer  as  CORBA  Client 

Now  we  want  to  study  the  effects  of  CORBA  clients  requesting  data  from  the  producer/con¬ 
sumer  application.  Again,  we  compare  different  techniques  for  decoupling  CORBA  and  the 
application’s  timing  behavior.  Since  varying  numbers  of  CORBA  clients  sending  requests  to 
our  producer/consumer  application  may  impose  unacceptably  high  loads,  we  must  develop  a 
scheme  for  call  admission,  which  controls  the  behavior  of  the  object  request  broker’s  dis¬ 
patcher. 

A  first  solution  to  the  potential  burstiness  problem  of  CORBA  requests  is  the  introduction  of  a 
data  cache  between  the  CORBA  server  and  the  producer/consumer  application.1  In  our  case, 
the  composite  object’s  shared  RT/NRT  variables,  with  its  weak  memory  consistency,  and  the 
periodic  handler  thread  implement  exactly  as  such  a  data  cache.  We  should  mention  that  cach¬ 
ing  does  not  completely  solve  the  problem  of  bursty  CORBA  requests:  sufficiently  high  num¬ 
bers  of  clients  still  can  create  unacceptably  high  loads  on  the  server’s  system. 

In  Figure  3-9,  we  compare  the  initial  case  where  the  CORBA  server  directly  accesses  the 
shared  memory  buffer  used  by  the  producer/consumer  with  a  variant  where  the  composite 
object  performs  caching.  We  have  run  our  experiments  for  different  numbers  of  CORBA  cli¬ 
ents.  Figure  3-9  depicts  the  cases  of  one,  two,  and  five  CORBA  clients  sending  requests  with  a 
frequency  of  10  Hz. 


1 .  In  this  report,  the  term  “burstiness”  is  used  to  refer  to  the  pattern  of  incoming  CORBA  requests. 
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Data  Caching  -  Communication  via  shared  memory 


initial  -  no  caching 
Variance:  (1  client)  0.067232 
(2  clients)  0.089120 
(5  clients)  0.128861 


0  SO  100  150  200  250  300  350  400  450  5QQ 

number  of  executions 


Caching  on  CORBA  side  of  Composite  Object 
Variance:  (1  client)  0.069507 
(2  clients)  0.077423 
(5  clients)  0.086160 


Figure  3-9:  Minimizing  Data  Accesses:  Effects  of  Caching 


In  the  left  diagram  of  Figure  3-9,  the  producer/consumer’s  periods  change  with  the  number  of 
active  CORBA  clients.  In  contrast,  as  shown  in  the  right  diagram,  caching  in  the  composite 
object  lowers  the  variations,  but  introduces  some  overhead  and  a  slight,  constant  increase  to 
the  producer/consumer’s  periods.  This  demonstrates  how  the  composite  objects  approach  can 
be  used  to  make  an  explicit  tradeoff  between  an  application’s  predictability  and  its  communi¬ 
cation  performance  and  overall  execution  time. 

The  scheduling  server  [Poize  96],  as  developed  earlier  in  context  of  the  SONiC  project  [Poize 
98],  represents  another  way  to  implement  call  admission  for  a  composite  object  acting  as  a 
CORBA  server.  Using  fixed-priority  scheduling  and  dynamic  changes  to  the  priority  of  a  cli¬ 
ent’s  tasks,  the  scheduling  server  implements  sharing  of  CPU  resources  on  an  a  priori  known 
basis.  The  scheduling  server’s  quantum  is  adjustable. 

For  the  diagrams  shown  in  Figure  3-10,  we  have  used  the  scheduling  server  to  restrict  the  CPU 
cycles  available  to  the  main  loop  of  the  CORBA  object  request  broker.  In  contrast  to  the  left 
diagram,  where  the  ORB  is  not  restricted  in  its  CPU  usage,  the  middle  and  right  diagrams 
show  cases  where  a  10  ms  quantum  is  allocated  to  CORBA,  once  every  50  ms  (middle  dia¬ 
gram),  or  once  every  80  ms  (right  diagram).  Thus,  the  remainder  of  the  scheduling  server’s 
period  of  40  ms  (middle  diagram)  and  70  ms  (right  diagram)  are  left  as  undisturbed  phases  for 
the  producer/consumer  application  on  the  otherwise  unloaded  computer. 
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Call  Admission  via  Scheduling  Server  -  Communication  via  pipes 


initial  -  no  Scheduling  Server 


Period:  50ms,  CORBA:  10ms  quantum  Period:  80ms,  CORBA:  10ms  quantum 


Variance:  (1  client)  0.072025  Variance:  (1  client)  0.077117  Variance:  {1  client)  0.074085 

(2  clients)  0.091340  /2  clients)  0.077483  (2  clients)  0.056264 

Hnim  (5  clients)  0.146772  tinms.  (5  clients)  0.088174  tmrns.  (5  clients)  0.059500 


Figure  3-10:  Restricting  the  ORB:  Call  Admission  via  Scheduling  Server 


The  scheduling  server’s  effect  is  quite  obvious.  In  contrast  to  the  first  case,  where  the  pro- 
ducer/consumer’s  periodicity  varies  largely  with  changing  CORBA  loads,  the  latter  two  cases 
show  much  more  stable  behavior.  Again,  we  see  a  tradeoff  between  predictable  behavior  and 
communication  latency;  the  length  of  the  producer/consumer’s  period  increases  when  we  do 
not  restrict  the  ORB  in  its  CPU  usage. 

Since  the  scheduling  server  suspends  the  object  request  broker’s  main  loop,  our  approach 
implements  true  call  admission.  Even  high  numbers  of  CORBA  clients  and  high  burstiness  of 
traffic  hardly  affect  the  producer/consumer’s  timing  behavior.  The  application  of  both  the 
composite  objects  approach  and  scheduling  server  concept  did  not  require  any  changes  to  the 
implementation  of  the  CORBA  object  request  broker  nor  the  operating  system’s  kernel. 
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Fault-Tolerance  Case  Studies 


4.1  Observable  Objects:  A  Technique  for  Robust 
CORBA  Clients 

We  have  developed  the  Observable  Object/Observer  technique  for  fault  tolerance.  Following  a 
primary/backup  approach  our  technique  allows  observation,  saving  of  state,  and  automatic 
restart  of  (distributed)  applications.  The  Observable  Object  base  class  can  be  seen  as  a  generic 
interface  for  supervision  of  critical  applications.  Derived  classes  may  implement  observation 
of  a  program  using  special,  application-dependent  programming  interfaces.  We  have  applied 
our  Observable  Object/Observer  technique  to  the  Netscape  Navigator  Web  browser  to  create  a 
fault-tolerant  Web  client. 

The  Observable  Object/Observer  architecture  is  based  on  CORBA.  Non-CORBA  legacy 
applications  may  take  advantage  of  our  framework  by  using  a  wrapper  approach.  To  encapsu¬ 
late  a  non-CORBA  application  in  a  wrapper,  the  application  must  offer  a  communication  or 
programming  interface  [e.g.,  UNIX  signals,  stdin/stdout,  IPC,  or  different  XI 1  mechanisms 
(e.g.,  properties)]  accessible  to  the  wrapper. 

Our  architecture  implements  a  generic  mechanism  for  monitoring  CORBA  objects.  This 
mechanism  is  used  for  increasing  the  reliability  of  the  monitored  objects.  The  Observer  can 
tolerate  two  classes  of  faults:  crash  faults  of  the  computer  nodes  and  crashes  of  the  monitored 
objects.  The  Observer  starts  monitored  objects  on  different  computation  nodes  in  a  replicated 
manner  according  to  the  primary/backup  principle.  The  Observer  checks  the  replies  of  primary 
and  backup  instances  to  periodic  ALIVE  messages.  The  primary  instance  regularly  saves  its 
state  on  stable  storage.  When  the  primary  fails,  the  Observer  selects  a  backup  for  reconfigura¬ 
tion  as  a  new  primary.  It  does  so  by  reading  the  last  saved  state. 

To  hike  advantage  of  our  Observable  Object/Observer  approach,  a  CORBA  object  has  to 
implement  the  observable  object  interface  shown  in  Figure  4-1. 

The  Observer  checks  availability  of  all  registered  observable  objects  by  calling  their  state() 
methods  in  user-specified  intervals.  The  object  reference  of  a  crashed  object  becomes  invalid. 
In  this  case,  the  CORBA  run-time  system  raises  an  exception.  Thus,  the  Observer  can  detect 
an  Observable  Object’s  failure  by  catching  this  exception. 
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interface  ObservableObject  { 
short  state(); 
short  id(); 

void  become_primary(in  short  o_id); 
oneway  void  shutdown(in  short  o_id); 

}; 


Figure  4-1:  Observable  Object  Interface:  COR  BA  IDL 


Factory  objects  are  used  to  create  new  backup  instances  of  our  service  in  case  of  crashes.  A 
factory’s  interface.  Observable  Factory,  is  derived  from  Observable  Object.  Thus,  the 
Observer  can  monitor  factory  objects.  Factories  are  assumed  to  be  well  tested  and  reliable; 
therefore  an  exception  resulting  from  an  invalid  reference  to  a  factory  object  is  interpreted  as 
an  indication  of  a  node  (computer)  failure. 

We  have  used  the  Observable  Object/Observer  architecture  to  increase  the  reliability  of  the 
Netscape  Navigator  Web  browser.  In  addition  to  a  graphical  user  interface,  most  browsers 
offer  a  programming  interface  for  controlling  the  browser  by  external  programs  [Zawinski  94]. 
This  functionality  is  used  by  a  wrapper,  which  implements  the  observable  object  interface.  A 
Web  browser  is  then  monitored  by  the  Observer  by  checking  the  availability  of  its  wrapper. 
Additionally,  the  state  of  the  browser  (current  URL)  is  determined  by  the  wrapper.  If  a  browser 
crashes,  the  wrapper  terminates  itself.  From  the  Observer’s  viewpoint,  a  browser  has  failed  if 
its  wrapper  is  no  longer  available.  We  assume  that  the  wrapper  will  not  fail  while  the  browser 
it  encapsulates  continues  to  run.  This  assumption  is  sound,  because  a  wrapper  is  a  small  pro¬ 
gram  that  can  be  tested  thoroughly. 

Figure  4-2  depicts  the  communication  structure  in  a  scenario  where  two  Observable  Objects 
acting  as  wrappers  for  Netscape  Navigator  processes  are  monitored  by  our  system.  Both  pri¬ 
mary  and  backup  are  connected  to  the  same  XI 1  display.  However,  the  backup’s  window  is 
unmapped  and  becomes  visible  only  when  there  is  a  crash  fault  of  the  primary. 

The  Observable  Object/Observer  concept  is  not  limited  to  CORBA.  It  can  be  adopted  for  other 
middleware  platforms  such  as  distributed  computing  environments  (DCEs)  or  the  Distributed 
Component  Model  (DCOM). 
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computer 

boundaries 


4.2  Consensus-Based  Responsive  Services 

In  many  cases,  it  is  desirable  to  predict  the  behavior  of  remote  services  invoked  via  CORBA. 
Unfortunately,  the  specification  of  parameters  for  service  execution  such  as  fault  models, 
acceptable  processor  utilization,  and  timing  behavior  is  beyond  the  scope  of  CORBA.  We  pro¬ 
pose  an  extension  of  our  previously  developed  CORE/SONiC  framework  [Malek  95,  Poize 
98]  for  responsive  computing  by  CORBA  interfaces  using  composite  objects  as  filtering 
bridges. 

In  the  CORE/SONiC  model,  several  parallel  tasks  may  run  on  a  number  of  interconnected 
computing  units.  SONiC  implements  an  object-based  distributed  shared  memory  system  to 
allow  for  communication  between  tasks.  Consensus  for  REsponsiveness  (CORE)  provides 
protocols,  services,  and  scheduling  strategies  at  the  microkernel  level  for  real-time  parallel 
computing,  even  in  the  presence  of  faults.  Consensus  protocols  are  used  to  detect  faults  and  to 
establish  a  system-wide  global  view  at  certain  points  of  program  execution.  This  serves  as  the 
basis  for  decisions  about  re-execution  and  coarse-grained  scheduling. 

CORE  provides  a  reliable  communication  subsystem  to  SONiC,  using  the  services  of  the 
underlying  Mach  microkernel  operating  system.  The  CORE/SONiC  scheduling  server  [Poize 
97,  Richling  97]  provides  predictable  access  to  workstation  resources  and  can  be  used  to 
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implement  real-time  scheduling  policies  such  as  rate  monotonic  scheduling  or  “earliest  dead¬ 
line  first”  on  a  standard  operating  system.  Composite  objects  provide  a  method  for  creating 
objects  that  interface  to  CORBA  and  are  simultaneously  capable  of  real-time  method  execu¬ 
tion.  We  use  the  composite  objects  approach  to  rebuild  the  concept  of  CORE/SONiC  in  a 
CORBA-based  environment.  A  group  of  composite  objects  distributed  over  the  nodes  of  a 
workstation  cluster  forms  the  execution  environment  for  responsive  services.  CORBA 
requests  are  the  basic  units  of  replication  and  scheduling  in  our  environment. 

Search  algorithms  benefit  highly  from  parallelization  and  load  partitioning  in  distributed  envi¬ 
ronments.  Here,  we  present  a  distributed  labyrinth  search  as  an  example  application.  Our 
application  demonstrates  structuring  of  a  responsive  service  and  employs  consensus  (group 
membership)  for  fault  tolerance.  Composite  objects  are  used  to  implement  timely  behavior  of 
the  consensus  protocol. 

The  scenario  depicted  in  Figure  4-3  shows  a  client  (Java  based)  that  generates  tasks — mazes 
with  one  entry  and  one  exit  point  in  our  case — with  different  structures.  These  tasks  are  repli¬ 
cated  and  sent  to  four  worker  nodes.  Each  worker  searches  a  subportion  of  the  maze.  Load  par¬ 
titioning  is  done  statically.  Each  search  step  is  reported  to  the  client  and  displayed  on  screen, 
using  different  colors  for  different  nodes. 

A  group  membership  protocol  is  run  among  the  worker  nodes.  Each  node  communicates  with 
its  two  left  neighbors  and  sends  periodic  alive  messages.  The  timing  behavior  of  the  protocol 
is  ensured  by  executing  it  within  real-time  threads  of  composite  objects.  The  scheduling  server 
is  used  to  allocate  a  fixed  percentage  of  CPU  cycles  (guarantee  slots)  to  the  search  processes. 
Our  responsive  service  tolerates  crash  faults  of  search  processes  and  processing  nodes.  The 
crash  fault  of  a  search  process  or  node  is  detected  by  missing  alive  messages.  In  this  case,  the 
communication  structure  is  reconfigured.  Furthermore,  a  crashed  node’s  left  neighbor  re-exe¬ 
cutes  the  crashed  node’s  task.  This  scheme  implements  graceful  degradation;  a  solution  to  the 
labyrinth  search  is  found  if  a  single  node  survives. 

To  demonstrate  the  behavior  described  above,  the  client  application  provides  a  user  interface 
to  send  kill-messages  to  a  particular  worker  node.  The  labyrinth  search  service  demonstrates 
responsiveness  (i.e.,  a  high  probability  to  deliver  a  timely  solution  under  a  given  load  and  fault 
hypothesis)  even  in  the  presence  of  faults. 
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5  Related  Work 


The  composite  objects  technique  can  be  used  as  a  bridge  between  CORBA’s  client/server 
model  and  execution  models  for  replicated,  fault-tolerant,  real-time  services.  Related  work 
describes  the  idea  of  providing  real  time  as  an  additional  feature  in  a  CORBA-compliant 
object  request  broker  (ORB)  implementation.  This  approach  has  been  proposed  by  the  OMG 
Real-Time  Special  Interest  Group. 

The  observable  object  interface  allows  for  implementation  of  robust  CORB  A  clients  following 
the  primary/backup  approach  to  fault  tolerance.  Related  work  describes  two  other  approaches 
toward  a  fault-tolerant  CORBA: 

1.  special  ORB  implementations,  which  may  take  advantage  of  underlying  fault-tolerant 
hardware 

2.  application  of  special  CORBA  services  for  fault  tolerance 

5.1  Real-Time  CORBA 

The  Object  Management  Group  (OMG)  founded  a  Real-Time  CORBA  Special  Interest  Group 
(SIG)  in  1996.  Since  then  OMG  has  solicited  technology  for  a  real-time  object  request  broker 
(ORB)  that  consists  of  fixed-priority  scheduling,  control  over  ORB  resources  for  end-to-end 
predictability,  and  flexible  communications.  The  request  for  proposal  (RFP)  for  the  fixed  pri¬ 
ority  version  of  Real-Time  CORBA  1.0  was  announced  in  September  1997  [OMG  97]. 

Work  at  the  University  of  Rhode  Island  and  the  MITRE  Corporation  deals  with  syntactical 
extensions  to  the  CORBA  interface  description  language  (IDL)  to  express  timing  constraints 
[Thuraisingham  96,  Wolfe  95].  “Timed  distributed  method  invocations”  are  identified  as  one 
necessary  feature  in  a  real-time  distributed  computing  environment  as  well  as  a  “global  time 
service,”  “real-time  scheduling  of  services,”  a  “global  priority  service,”  and  “bounded  mes¬ 
sage  latency.”  The  “affected  set  priority  ceiling  protocol”  as  a  combination  of  semantic  lock¬ 
ing  and  priority  ceiling  techniques  has  been  proposed  for  concurrency  control  in  real-time 
object-oriented  systems  [Squadrito  98]. 

TAO  is  an  innovative  work  on  Real-Time  CORBA,  where  fixed-priority  real-time  scheduling 
is  tightly  integrated  into  the  system  [Schmidt  97,  Schmidt  98],  The  main  goal  of  this  work  is  to 
provide  end-to-end  quality  of  service  for  CORBA-based  applications.  A  list  of  requirements 
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for  object  request  broker  implementations  is  presented;  among  them  are  resource  reservation 
protocols,  optimized  real-time  communication  protocols,  and  a  real-time  object  adapter.  TAO, 
however,  focuses  on  completely  new  CORBA-based  real-time  systems,  rather  than  interfacing 
existing  real-time  systems  with  CORBA. 


5.2  Fault-Tolerant  CORBA 

The  idea  of  providing  fault  tolerance  as  an  additional  feature  to  CORBA  implementations  has 
been  the  focus  for  several  research  activities  within  the  last  few  years.  With  the  request  for 
proposal  issued  in  April  1998  [OMG  98],  OMG  is  seeking  to  incorporate  existing  approaches 
for  software  fault  tolerance  into  future  versions  of  CORBA. 

Electra  is  a  CORBA  ORB  implementation  for  reliable,  distributed  services  [Mafefis  94].  Elec- 
tra  extends  the  CORBA  specification  and  provides  group  communication  mechanisms,  reli¬ 
able  multicasts,  and  object  replication.  The  Electra-ORB  uses  services  from  the  underlying 
ISIS  systems  [Birman  93,  Renesse  94], 

ORBIX+ISIS  is  an  extension  of  the  commercial  ORBIX  [IONA]  ORB  implementation  that 
introduces  concepts  such  as  object  groups  and  group  communication  based  on  the  ISIS  toolkit 
[Birman  93].  Again,  the  CORBA  standard  has  been  extended  to  allow  for  introduction  of 
fault-tolerance  measures.  The  programmer  must  use  special  coding  techniques  to  use  the  fault- 
tolerance  features  of  ORBIX+ISIS. 

Phoinix  [Chang  97]  allows  for  implementation  of  server  objects  following  the  primary/backup 
approach  for  fault  tolerance.  In  contrast  to  changing  the  ORB,  Phoinix  defines  a  new  service 
as  part  of  the  Object  Management  Architecture  (OMA).  Using  this  service,  a  primary  object’s 
state  can  be  periodically  transferred  into  the  corresponding  backup  object.  Clients  communi¬ 
cate  solely  with  the  primary  object  and  have  to  detect  failures  themselves.  Extended  skeletons, 
stubs,  and  libraries  are  provided  to  make  those  fault-tolerance  services  accessible. 
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6  Conclusions  and  Future  Work 


There  is  an  urgent  need  to  support  heterogeneity  and  openness  in  today’s  COTS-based  distrib¬ 
uted  computing  environments  and  to  enhance  the  computing  environments  by  quality-'of-ser- 
vice  attributes  such  as  responsiveness.  Although  CORBA  is  a  good  candidate  for  support  of 
heterogeneity  and  openness,  it  does  not  allow  specification  of  resource,  scheduling,  and  timing 
requirements;  fault  models;  or  measures.  Based  on  the  composite  objects  approach,  we  have 
developed  building  blocks  that  integrate  proven  techniques  for  predictable  computing  with 
CORBA. 

We  have  discussed  the  composite  objects  approach  for  the  integration  of  CORBA  with  real¬ 
time  computing.  Data  replication  and  weak  memory  consistency  have  been  discussed  as  key 
concepts  for  implementing  this  approach  and  for  decoupling  CORBA  and  real-time  comput¬ 
ing.  We  have  described  implementation  alternatives  for  the  composite  objects  approach  and 
have  presented  measurements  for  the  overhead  imposed  by  our  implementation  of  this 
approach. 

The  producer/consumer/viewer  and  “balancing  robots”  example  scenarios  study  the  effects  of 
bridging  legacy  applications  to  CORBA  via  composite  objects.  Our  measurements  indicate 
that  the  composite  objects  approach  provides  a  promising  way  to  retain  an  application’s  pre¬ 
dictable  timing  behavior,  even  when  communicating  via  CORBA.  Furthermore,  using  the 
scheduling  server  developed  earlier,  we  have  discussed  and  demonstrated  how  call  admission, 
as  a  technique  for  bounding  the  ORB’s  resource  utilization,  can  be  implemented  without 
changes  to  the  CORBA  implementation  and  the  operating  system’s  kernel. 

We  have  presented  the  CORBA-based  observer  approach,  a  generic  solution  for  implementing 
reliable  applications  (fault-tolerant  Netscape  Web  client  in  our  case).  Bridging  the  gap 
between  (legacy)  real-time  fault-tolerant  computing  systems  and  CORBA-based  clients  is  a 
major  precondition  for  remote  operations  in  many  application  domains  such  as  banking,  medi¬ 
cine,  manufacturing,  and  booking  systems. 

The  composite  objects  approach  represents  a  viable  technique  to  make  CORBA-based  cluster 
computing  predictable  in  its  resource  utilization  and  timing  behavior.  Future  work  will  include 
detailed  studies  of  alternative  implementations  for  composite  objects  on  the  operating  systems 
rtLinux,  Solaris,  and  Windows  NT. 
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objects  are  also  applicable  to  platforms  like  the  Distributed  Component  Model  (DCOM)  and  distributed 
computing  environments  (DCEs).  Key  concepts  in  Humboldt’s  approach  are  analytic  redundancy, 
noninterference,  interoperability,  and  adaptive  abstraction.  These  concepts  originated  in  SEI  work  on  the 
Simplex  architecture  and  have  been  reapplied  to  extend  the  reach  of  commercial  off-the-shelf  (COTS) 
software  technologies  into  demanding  application  settings  (such  as  those  found  in  military  and  industrial 
applications).  Here,  we  discuss  building  blocks  and  techniques  for  fault-tolerant,  real-time  applications  based 
on  CORBA. 
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